{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference PyTorch Bert Model with ONNX Runtime on GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, you'll learn how to load a Bert model from PyTorch, convert it to ONNX, and inference it for high performance using ONNX Runtime and NVIDIA GPU. In the following sections, we are going to use the Bert model trained with Stanford Question Answering Dataset (SQuAD) dataset as an example. Bert SQuAD model is used in question answering scenarios, where the answer to every question is a segment of text from the corresponding reading passage, or the question might be unanswerable.\n",
    "\n",
    "This notebook is for GPU inference. For CPU inference, please look at another notebook [Inference PyTorch Bert Model with ONNX Runtime on CPU](PyTorch_Bert-Squad_OnnxRuntime_CPU.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Prerequisites ##\n",
    "It requires your machine to have a GPU, and a python environment with [PyTorch](https://pytorch.org/) installed before running this notebook.\n",
    "\n",
    "#### GPU Environment Setup using AnaConda\n",
    "\n",
    "First, we install [AnaConda](https://www.anaconda.com/distribution/) in a target machine and open an AnaConda prompt window when it is done. Then run the following commands to create a conda environment. This notebook is tested with PyTorch 1.5.0 and OnnxRuntime 1.3.0.\n",
    "\n",
    "```console\n",
    "conda create -n gpu_env python=3.7\n",
    "conda activate gpu_env\n",
    "conda install pytorch torchvision cudatoolkit=10.1 -c pytorch\n",
    "conda install -c anaconda ipykernel\n",
    "conda install -c conda-forge ipywidgets\n",
    "python -m ipykernel install --user --name=gpu_env_py37\n",
    "jupyter notebook\n",
    "```\n",
    "Finally, launch Jupyter Notebook and you can choose gpu_env_py37 as kernel to run this notebook.\n",
    "\n",
    "Onnxruntime-gpu need specified version of CUDA and cuDNN. You can find the corresponding version in [requirements](https://github.com/microsoft/onnxruntime/tree/rel-1.3.0#system-requirements). If the version is different from above cudatoolkit version, you have to install them separately, and add their bin directories to PATH environment variable (See [CUDA and cuDNN Path](#CUDA-and-cuDNN-Path) below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Skipping onnxruntime-gpu as it is not installed.\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip uninstall --quiet --yes onnxruntime-gpu\n",
    "!{sys.executable} -m pip install --quiet onnxruntime-gpu\n",
    "!{sys.executable} -m pip install --quiet --upgrade transformers\n",
    "!{sys.executable} -m pip install --quiet --upgrade onnxconverter_common\n",
    "!{sys.executable} -m pip install --quiet --upgrade onnxruntime-tools\n",
    "!{sys.executable} -m pip install --quiet wget netron pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load Pretrained Bert model ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We begin by downloading the SQuAD data file and store them in the specified location. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "cache_dir = \"./squad\"\n",
    "if not os.path.exists(cache_dir):\n",
    "    os.makedirs(cache_dir)\n",
    "\n",
    "predict_file_url = \"https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json\"\n",
    "predict_file = os.path.join(cache_dir, \"dev-v1.1.json\")\n",
    "if not os.path.exists(predict_file):\n",
    "    import wget\n",
    "    print(\"Start downloading predict file.\")\n",
    "    wget.download(predict_file_url, predict_file)\n",
    "    print(\"Predict file downloaded.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first define some constant variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Whether allow overwriting existing ONNX model and download the latest script from GitHub\n",
    "enable_overwrite = True\n",
    "\n",
    "# Total samples to inference, so that we can get average latency\n",
    "total_samples = 1000\n",
    "\n",
    "# ONNX opset version\n",
    "opset_version=11"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify some model configuration variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For fine-tuned large model, the model name is \"bert-large-uncased-whole-word-masking-finetuned-squad\". Here we use bert-base for demo.\n",
    "model_name_or_path = \"bert-base-cased\"\n",
    "max_seq_length = 128\n",
    "doc_stride = 128\n",
    "max_query_length = 64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start to load model from pretrained. This step could take a few minutes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 48/48 [00:04<00:00, 11.28it/s]\n",
      "convert squad examples to features: 100%|██████████| 1000/1000 [00:09<00:00, 102.15it/s]\n",
      "add example index and unique id: 100%|██████████| 1000/1000 [00:00<00:00, 161306.98it/s]\n"
     ]
    }
   ],
   "source": [
    "# The following code is adapted from HuggingFace transformers\n",
    "# https://github.com/huggingface/transformers/blob/master/examples/run_squad.py\n",
    "\n",
    "from transformers import (BertConfig, BertForQuestionAnswering, BertTokenizer)\n",
    "\n",
    "# Load pretrained model and tokenizer\n",
    "config_class, model_class, tokenizer_class = (BertConfig, BertForQuestionAnswering, BertTokenizer)\n",
    "config = config_class.from_pretrained(model_name_or_path, cache_dir=cache_dir)\n",
    "tokenizer = tokenizer_class.from_pretrained(model_name_or_path, do_lower_case=True, cache_dir=cache_dir)\n",
    "model = model_class.from_pretrained(model_name_or_path,\n",
    "                                    from_tf=False,\n",
    "                                    config=config,\n",
    "                                    cache_dir=cache_dir)\n",
    "# load some examples\n",
    "from transformers.data.processors.squad import SquadV1Processor\n",
    "\n",
    "processor = SquadV1Processor()\n",
    "examples = processor.get_dev_examples(None, filename=predict_file)\n",
    "\n",
    "from transformers import squad_convert_examples_to_features\n",
    "features, dataset = squad_convert_examples_to_features( \n",
    "            examples=examples[:total_samples], # convert enough examples for this notebook\n",
    "            tokenizer=tokenizer,\n",
    "            max_seq_length=max_seq_length,\n",
    "            doc_stride=doc_stride,\n",
    "            max_query_length=max_query_length,\n",
    "            is_training=False,\n",
    "            return_dataset='pt'\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Export the loaded model ##\n",
    "Once the model is loaded, we can export the loaded PyTorch model to ONNX."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model exported at  ./onnx/bert-base-cased-squad_opset11.onnx\n"
     ]
    }
   ],
   "source": [
    "output_dir = \"./onnx\"\n",
    "if not os.path.exists(output_dir):\n",
    "    os.makedirs(output_dir)   \n",
    "export_model_path = os.path.join(output_dir, 'bert-base-cased-squad_opset{}.onnx'.format(opset_version))\n",
    "\n",
    "import torch\n",
    "use_gpu = torch.cuda.is_available()\n",
    "device = torch.device(\"cuda\" if use_gpu else \"cpu\")\n",
    "\n",
    "# Get the first example data to run the model and export it to ONNX\n",
    "data = dataset[0]\n",
    "inputs = {\n",
    "    'input_ids':      data[0].to(device).reshape(1, max_seq_length),\n",
    "    'attention_mask': data[1].to(device).reshape(1, max_seq_length),\n",
    "    'token_type_ids': data[2].to(device).reshape(1, max_seq_length)\n",
    "}\n",
    "\n",
    "# Set model to inference mode, which is required before exporting the model because some operators behave differently in \n",
    "# inference and training mode.\n",
    "model.eval()\n",
    "model.to(device)\n",
    "\n",
    "if enable_overwrite or not os.path.exists(export_model_path):\n",
    "    with torch.no_grad():\n",
    "        symbolic_names = {0: 'batch_size', 1: 'max_seq_len'}\n",
    "        torch.onnx.export(model,                                            # model being run\n",
    "                          args=tuple(inputs.values()),                      # model input (or a tuple for multiple inputs)\n",
    "                          f=export_model_path,                              # where to save the model (can be a file or file-like object)\n",
    "                          opset_version=opset_version,                      # the ONNX version to export the model to\n",
    "                          do_constant_folding=True,                         # whether to execute constant folding for optimization\n",
    "                          input_names=['input_ids',                         # the model's input names\n",
    "                                       'input_mask', \n",
    "                                       'segment_ids'],\n",
    "                          output_names=['start', 'end'],                    # the model's output names\n",
    "                          dynamic_axes={'input_ids': symbolic_names,        # variable length axes\n",
    "                                        'input_mask' : symbolic_names,\n",
    "                                        'segment_ids' : symbolic_names,\n",
    "                                        'start' : symbolic_names,\n",
    "                                        'end' : symbolic_names})\n",
    "        print(\"Model exported at \", export_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. PyTorch Inference ##\n",
    "Use PyTorch to evaluate an example input for comparison purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PyTorch cuda Inference time = 16.57 ms\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "# Measure the latency. It is not accurate using Jupyter Notebook, it is recommended to use standalone python script.\n",
    "latency = []\n",
    "with torch.no_grad():\n",
    "    for i in range(total_samples):\n",
    "        data = dataset[i]\n",
    "        inputs = {\n",
    "            'input_ids':      data[0].to(device).reshape(1, max_seq_length),\n",
    "            'attention_mask': data[1].to(device).reshape(1, max_seq_length),\n",
    "            'token_type_ids': data[2].to(device).reshape(1, max_seq_length)\n",
    "        }\n",
    "        start = time.time()\n",
    "        outputs = model(**inputs)\n",
    "        latency.append(time.time() - start)\n",
    "print(\"PyTorch {} Inference time = {} ms\".format(device.type, format(sum(latency) * 1000 / len(latency), '.2f')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Inference ONNX Model with ONNX Runtime ##\n",
    "\n",
    "### CUDA and cuDNN Path\n",
    "onnxruntime-gpu has dependency on [CUDA](https://developer.nvidia.com/cuda-downloads) and [cuDNN](https://developer.nvidia.com/cudnn):\n",
    "\n",
    "* [onnxruntime-gpu v1.3.0](https://github.com/microsoft/onnxruntime/tree/rel-1.3.0#system-requirements) requires CUDA Runtime 10.1 and CUDNN 7.6.5.\n",
    "* [onnxruntime-gpu v1.2.0](https://github.com/microsoft/onnxruntime/releases/tag/v1.2.0) requires CUDA Runtime 10.1 and CUDNN 7.6.5.\n",
    "\n",
    "During installing PyTorch 1.5, we installed cudatoolkit 10.1.243 in this conda environment. That shall be good for onnxruntime-gpu 1.3.0 in Jupyter Notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change to True when onnxruntime (like onnxruntime-gpu 1.0.0 ~ 1.1.2) cannot be imported.\n",
    "add_cuda_path = False\n",
    "\n",
    "if add_cuda_path:\n",
    "    # Add path of CUDA 10.0 and CUDNN 7.6 for onnxruntime-gpu 1.0.0 ~ 1.1.2\n",
    "    cuda_dir = 'D:/NVidia/CUDA/v10.1/bin'\n",
    "    cudnn_dir = 'D:/NVidia/CUDA/v10.1/bin'\n",
    "    if not (os.path.exists(cuda_dir) and os.path.exists(cudnn_dir)):\n",
    "        raise ValueError(\"Please specify correct path for CUDA and cuDNN. Otherwise onnxruntime cannot be imported.\")\n",
    "    else:\n",
    "        if cuda_dir == cudnn_dir:\n",
    "            os.environ[\"PATH\"] = cuda_dir + ';' + os.environ[\"PATH\"]\n",
    "        else:\n",
    "            os.environ[\"PATH\"] = cuda_dir + ';' + cudnn_dir + ';' + os.environ[\"PATH\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OpenMP Environment Variable\n",
    "\n",
    "OpenMP environment variables are optional for GPU inference of standard Bert model. It has little performance impact on Bert model since most nodes are executed in GPU. \n",
    "\n",
    "You can find the best setting based on [Performance Test Tool](#Performance-Test-Tool) result in later part of this notebook.\n",
    "\n",
    "**Attention: Setting environment variables shall be done before importing onnxruntime**. Otherwise, they might not take effect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optional. You can change them according to Performance Test Tool result.\n",
    "#os.environ[\"OMP_NUM_THREADS\"] = '1'\n",
    "#os.environ[\"OMP_WAIT_POLICY\"] = 'PASSIVE'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to inference the model with ONNX Runtime."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OnnxRuntime gpu Inference time = 4.43 ms\n"
     ]
    }
   ],
   "source": [
    "import psutil\n",
    "import onnxruntime\n",
    "import numpy\n",
    "\n",
    "assert 'CUDAExecutionProvider' in onnxruntime.get_available_providers()\n",
    "device_name = 'gpu'\n",
    "\n",
    "sess_options = onnxruntime.SessionOptions()\n",
    "\n",
    "# Optional: store the optimized graph and view it using Netron to verify that model is fully optimized.\n",
    "# Note that this will increase session creation time so enable it for debugging only.\n",
    "sess_options.optimized_model_filepath = os.path.join(output_dir, \"optimized_model_{}.onnx\".format(device_name))\n",
    "\n",
    "# Please change the value according to best setting in Performance Test Tool result.\n",
    "sess_options.intra_op_num_threads=psutil.cpu_count(logical=True)\n",
    "\n",
    "session = onnxruntime.InferenceSession(export_model_path, sess_options)\n",
    "\n",
    "latency = []\n",
    "for i in range(total_samples):\n",
    "    data = dataset[i]\n",
    "    # TODO: use IO Binding (see https://github.com/microsoft/onnxruntime/pull/4206) to improve performance.\n",
    "    ort_inputs = {\n",
    "        'input_ids':  data[0].cpu().reshape(1, max_seq_length).numpy(),\n",
    "        'input_mask': data[1].cpu().reshape(1, max_seq_length).numpy(),\n",
    "        'segment_ids': data[2].cpu().reshape(1, max_seq_length).numpy()\n",
    "    }\n",
    "    start = time.time()\n",
    "    ort_outputs = session.run(None, ort_inputs)\n",
    "    latency.append(time.time() - start)\n",
    "    \n",
    "print(\"OnnxRuntime {} Inference time = {} ms\".format(device_name, format(sum(latency) * 1000 / len(latency), '.2f')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compare the output of PyTorch and ONNX Runtime. We can see some results are not close. It is because ONNX Runtime uses some approximation in CUDA optimization. Based on our evaluation on SQuAD data set, F1 score is on par for models before and after optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** Verifying correctness *****\n",
      "PyTorch and ONNX Runtime output 0 are close: True\n",
      "maximum_diff=9.499490261077881e-07 average_diff=1.4225952327251434e-07\n",
      "PyTorch and ONNX Runtime output 1 are close: True\n",
      "maximum_diff=6.92903995513916e-07 average_diff=1.2441887520253658e-07\n"
     ]
    }
   ],
   "source": [
    "print(\"***** Verifying correctness *****\")\n",
    "for i in range(2):    \n",
    "    print('PyTorch and ONNX Runtime output {} are close:'.format(i), numpy.allclose(ort_outputs[i], outputs[i].cpu(), rtol=1e-02, atol=1e-02))\n",
    "    diff = ort_outputs[i] - outputs[i].cpu().numpy()\n",
    "    max_diff = numpy.max(numpy.abs(diff))\n",
    "    avg_diff = numpy.average(numpy.abs(diff))\n",
    "    print(f'maximum_diff={max_diff} average_diff={avg_diff}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference with Actual Sequence Length\n",
    "Note that ONNX model is exported using dynamic length axis. It is recommended to use actual sequence input without padding instead of fixed length input for best performance. Let's see how it can be applied to this model.\n",
    "\n",
    "From an example input below, we can see zero padding at the end of each sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[  101,  1293,  1242,  2557,  1127,  1226,  1104,  1103,  3613, 16429,\n",
       "           5235,   136,   102,  3613, 16429,  5988,   170,   107,  1353,  1671,\n",
       "           1992,  1342,   107,  5235,   117,  1107,  1134,  1473,  3683,  3538,\n",
       "           1125,   170,  1476,   118,  1248,  2595,  4086,  1714,  1104,  2965,\n",
       "          15897,  1104,  3613, 16429,   119,  1473,  3683,  3538,  3222,  1149,\n",
       "           2551,  1168, 23759,  1116,  1121,  1506,  1103, 10280,  2231,  1111,\n",
       "           1103,  1714, 16355,   119,   102,     0,     0,     0,     0,     0,\n",
       "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "              0,     0,     0,     0,     0,     0,     0,     0]],\n",
       "        device='cuda:0'),\n",
       " 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0,\n",
       "          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "          0, 0, 0, 0, 0, 0, 0, 0]], device='cuda:0'),\n",
       " 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0,\n",
       "          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "          0, 0, 0, 0, 0, 0, 0, 0]], device='cuda:0')}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# An example input (we can see padding). From attention_mask, we can deduce the actual length.\n",
    "inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The original sequence length is 128. After removing paddings, the sequence length is reduced. Input with smaller sequence length need less computation, thus we can see there is improvement on inference latency. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average length 101\n",
      "OnnxRuntime gpu Inference time with actual sequence length = 4.23 ms\n"
     ]
    }
   ],
   "source": [
    "import statistics\n",
    "\n",
    "latency = []\n",
    "lengths = []\n",
    "for i in range(total_samples):\n",
    "    data = dataset[i]\n",
    "    # Instead of using fixed length (128), we can use actual sequence length (less than 128), which helps to get better performance.\n",
    "    actual_sequence_length = sum(data[1].numpy())\n",
    "    lengths.append(actual_sequence_length)\n",
    "    opt_inputs = {\n",
    "        'input_ids':  data[0].numpy()[:actual_sequence_length].reshape(1, actual_sequence_length),\n",
    "        'input_mask': data[1].numpy()[:actual_sequence_length].reshape(1, actual_sequence_length),\n",
    "        'segment_ids': data[2].numpy()[:actual_sequence_length].reshape(1, actual_sequence_length)\n",
    "    }\n",
    "    start = time.time()\n",
    "    opt_outputs = session.run(None, opt_inputs)\n",
    "    latency.append(time.time() - start)\n",
    "print(\"Average length\", statistics.mean(lengths))\n",
    "print(\"OnnxRuntime {} Inference time with actual sequence length = {} ms\".format(device_name, format(sum(latency) * 1000 / len(latency), '.2f')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare the output and see whether the results are close.\n",
    "\n",
    "**Note**: Need end-to-end evaluation on performance and accuracy if you use this strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** Comparing results with/without paddings *****\n",
      "Output 0 are close: True\n",
      "Output 1 are close: True\n"
     ]
    }
   ],
   "source": [
    "print(\"***** Comparing results with/without paddings *****\")\n",
    "for i in range(2):\n",
    "    print('Output {} are close:'.format(i), numpy.allclose(opt_outputs[i], ort_outputs[i][:,:len(opt_outputs[i][0])], rtol=1e-03, atol=1e-03))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Offline Optimization and Test Tools\n",
    "\n",
    "It is recommended to try [OnnxRuntime Transformer Model Optimization Tool](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers) on the exported ONNX models. It could help verify whether the model can be fully optimized, and get performance test results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transformer Optimizer\n",
    "\n",
    "Although OnnxRuntime could optimize Bert model exported by PyTorch. Sometime, model cannot be fully optimized due to different reasons:\n",
    "* A new subgraph pattern is generated by new version of export tool, and the pattern is not covered by older version of OnnxRuntime. \n",
    "* The exported model uses dynamic axis and this makes it harder for shape inference of the graph. That blocks some optimization to be applied.\n",
    "* Some optimization is better to be done offline. Like change input tensor type from int64 to int32 to avoid extra Cast nodes, or convert model to float16 to achieve better performance in V100 or T4 GPU.\n",
    "\n",
    "We have python script **optimizer.py**, which is more flexible in graph pattern matching and model conversion (like float32 to float16). You can also use it to verify whether a Bert model is fully optimized.\n",
    "\n",
    "In this example, we can see that it introduces optimization that is not provided by onnxruntime: SkipLayerNormalization and bias fusion, which is not fused in OnnxRuntime due to shape inference as mentioned.\n",
    "\n",
    "It will also tell whether the model is fully optimized or not. If not, that means you might need change the script to fuse some new pattern of subgraph.\n",
    "\n",
    "Example Usage:\n",
    "```\n",
    "from onnxruntime_tools import optimizer\n",
    "optimized_model = optimizer.optimize_model(export_model_path, model_type='bert', num_heads=12, hidden_size=768)\n",
    "optimized_model.save_model_to_file(optimized_model_path)\n",
    "```\n",
    "\n",
    "You can also use optimizer_cli like the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Float32 Model\n",
    "Let us optimize the ONNX model using the script. The first example will output model with float32 to store weights. This is the choice for most GPUs without Tensor Core.\n",
    "\n",
    "If your GPU (like V100 or T4) has Tensor Core, jump to [Float16 Model](#6.-Model-Optimization-with-Float16) section since that will give you better performance than Float32 model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "optimize_by_onnxruntime: Save optimized model by onnxruntime to ./onnx/bert-base-cased-squad_opset11_o1_cpu.onnx\n",
      "               apply: Fused LayerNormalization count: 25\n",
      "               apply: Fused Gelu count: 12\n",
      "               apply: Fused SkipLayerNormalization count: 25\n",
      "               apply: Fused Attention count: 12\n",
      "         prune_graph: Graph pruned: 0 inputs, 0 outputs and 5 nodes are removed\n",
      "               apply: Fused EmbedLayerNormalization(with mask) count: 1\n",
      "         prune_graph: Graph pruned: 0 inputs, 0 outputs and 12 nodes are removed\n",
      "         prune_graph: Graph pruned: 0 inputs, 0 outputs and 0 nodes are removed\n",
      "               apply: Fused BiasGelu count: 12\n",
      "               apply: Fused SkipLayerNormalization(add bias) count: 24\n",
      "            optimize: opset verion: 11\n",
      "  save_model_to_file: Output model to ./onnx/bert-base-cased-squad_opt_gpu_fp32.onnx\n",
      "get_fused_operator_statistics: Optimized operators:{'EmbedLayerNormalization': 1, 'Attention': 12, 'Gelu': 0, 'FastGelu': 0, 'BiasGelu': 12, 'LayerNormalization': 0, 'SkipLayerNormalization': 24}\n",
      "                main: The model has been fully optimized.\n"
     ]
    }
   ],
   "source": [
    "optimized_fp32_model_path = './onnx/bert-base-cased-squad_opt_{}_fp32.onnx'.format('gpu' if use_gpu else 'cpu')\n",
    "\n",
    "!python -m onnxruntime_tools.optimizer_cli --input $export_model_path --output $optimized_fp32_model_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optimized Graph\n",
    "We can open the optimized model using [Netron](https://github.com/lutzroeder/netron) to visualize.\n",
    "\n",
    "The graph is like the following:\n",
    "<img src='images/optimized_bert_gpu.png'>\n",
    "\n",
    "Sometime, optimized graph is slightly different. For example, FastGelu is replaced by BiasGelu for CPU inference; When the option --input_int32 is used, Cast nodes for inputs are removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import netron\n",
    "\n",
    "# change it to True if want to view the optimized model in browser\n",
    "enable_netron = False\n",
    "if enable_netron:\n",
    "    # If you encounter error \"access a socket in a way forbidden by its access permissions\", install Netron as standalone application instead.\n",
    "    netron.start(optimized_fp32_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance Test Tool\n",
    "\n",
    "The following will create 1000 random inputs of batch_size 1 and sequence length 128, then measure the average latency and throughput numbers.\n",
    "\n",
    "Note that the test uses fixed sequence length. If you use [dynamic sequence length](#Inference-with-Actual-Sequence-Length), actual performance depends on the distribution of sequence length.\n",
    "\n",
    "**Attention**: Latency numbers from Jupyter Notebook are not accurate. See [Attional Info](#7.-Additional-Info) for more info."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test setting TestSetting(batch_size=1, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=None, omp_wait_policy=None, intra_op_num_threads=None, seed=3, verbose=False, inclusive=False, extra_latency=True)\n",
      "Generating 1000 samples for batch_size=1 sequence_length=128\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=0,OMP_NUM_THREADS=,OMP_WAIT_POLICY=,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 4.92 ms, Throughput = 203.24 QPS\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 4.90 ms, Throughput = 203.88 QPS\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=1,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 5.07 ms, Throughput = 197.16 QPS\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 4.82 ms, Throughput = 207.33 QPS\n",
      "skip duplicated test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "skip duplicated test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=1,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=1,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 4.93 ms, Throughput = 202.92 QPS\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 4.91 ms, Throughput = 203.55 QPS\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 4.88 ms, Throughput = 204.90 QPS\n",
      "Test summary is saved to onnx/perf_results_GPU_B1_S128_20200617-232134.txt\n"
     ]
    }
   ],
   "source": [
    "GPU_OPTION = '--use_gpu' if use_gpu else ''\n",
    "\n",
    "!python -m onnxruntime_tools.transformers.bert_perf_test --model $optimized_fp32_model_path --batch_size 1 --sequence_length 128 --samples 1000 --test_times 1 --inclusive --all $GPU_OPTION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the summary file and take a look. Note that blank value in OMP_NUM_THREADS or OMP_WAIT_POLICY means the environment variable does not exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Float32 model perf results from ./onnx/perf_results_GPU_B1_S128_20200617-232134.txt\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Latency(ms)</th>\n",
       "      <th>Latency_P50</th>\n",
       "      <th>Latency_P75</th>\n",
       "      <th>Latency_P90</th>\n",
       "      <th>Latency_P95</th>\n",
       "      <th>Latency_P99</th>\n",
       "      <th>Throughput(QPS)</th>\n",
       "      <th>intra_op_num_threads</th>\n",
       "      <th>OMP_NUM_THREADS</th>\n",
       "      <th>OMP_WAIT_POLICY</th>\n",
       "      <th>contiguous</th>\n",
       "      <th>warmup</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.82</td>\n",
       "      <td>4.53</td>\n",
       "      <td>4.57</td>\n",
       "      <td>5.15</td>\n",
       "      <td>7.25</td>\n",
       "      <td>8.75</td>\n",
       "      <td>207.33</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.88</td>\n",
       "      <td>4.54</td>\n",
       "      <td>4.58</td>\n",
       "      <td>6.47</td>\n",
       "      <td>7.13</td>\n",
       "      <td>8.68</td>\n",
       "      <td>204.90</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>PASSIVE</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.90</td>\n",
       "      <td>4.54</td>\n",
       "      <td>4.57</td>\n",
       "      <td>6.16</td>\n",
       "      <td>7.64</td>\n",
       "      <td>8.82</td>\n",
       "      <td>203.88</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>PASSIVE</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.91</td>\n",
       "      <td>4.55</td>\n",
       "      <td>4.59</td>\n",
       "      <td>6.70</td>\n",
       "      <td>7.43</td>\n",
       "      <td>8.78</td>\n",
       "      <td>203.55</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.92</td>\n",
       "      <td>4.57</td>\n",
       "      <td>4.60</td>\n",
       "      <td>6.50</td>\n",
       "      <td>7.82</td>\n",
       "      <td>8.90</td>\n",
       "      <td>203.24</td>\n",
       "      <td>0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4.93</td>\n",
       "      <td>4.55</td>\n",
       "      <td>4.59</td>\n",
       "      <td>6.66</td>\n",
       "      <td>7.57</td>\n",
       "      <td>8.80</td>\n",
       "      <td>202.92</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>PASSIVE</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>5.07</td>\n",
       "      <td>4.56</td>\n",
       "      <td>4.61</td>\n",
       "      <td>7.19</td>\n",
       "      <td>8.11</td>\n",
       "      <td>9.01</td>\n",
       "      <td>197.16</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Latency(ms)  Latency_P50  Latency_P75  Latency_P90  Latency_P95  \\\n",
       "0         4.82         4.53         4.57         5.15         7.25   \n",
       "1         4.88         4.54         4.58         6.47         7.13   \n",
       "2         4.90         4.54         4.57         6.16         7.64   \n",
       "3         4.91         4.55         4.59         6.70         7.43   \n",
       "4         4.92         4.57         4.60         6.50         7.82   \n",
       "5         4.93         4.55         4.59         6.66         7.57   \n",
       "6         5.07         4.56         4.61         7.19         8.11   \n",
       "\n",
       "   Latency_P99  Throughput(QPS)  intra_op_num_threads OMP_NUM_THREADS  \\\n",
       "0         8.75           207.33                     1              12   \n",
       "1         8.68           204.90                    12              12   \n",
       "2         8.82           203.88                     1              12   \n",
       "3         8.78           203.55                    12              12   \n",
       "4         8.90           203.24                     0                   \n",
       "5         8.80           202.92                    12               1   \n",
       "6         9.01           197.16                    12               1   \n",
       "\n",
       "  OMP_WAIT_POLICY contiguous  warmup  \n",
       "0          ACTIVE       None    True  \n",
       "1         PASSIVE       None    True  \n",
       "2         PASSIVE       None    True  \n",
       "3          ACTIVE       None    True  \n",
       "4                       None    True  \n",
       "5         PASSIVE       None    True  \n",
       "6          ACTIVE       None    True  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import glob     \n",
    "import pandas\n",
    "latest_result_file = max(glob.glob(\"./onnx/perf_results_GPU_B1_S128_*.txt\"), key=os.path.getmtime)\n",
    "result_data = pandas.read_table(latest_result_file, converters={'OMP_NUM_THREADS': str, 'OMP_WAIT_POLICY':str})\n",
    "print(\"Float32 model perf results from\", latest_result_file)\n",
    "# Remove some columns that have same values for all rows.\n",
    "columns_to_remove = ['model', 'graph_optimization_level', 'batch_size', 'sequence_length', 'test_cases', 'test_times', 'use_gpu']\n",
    "result_data.drop(columns_to_remove, axis=1, inplace=True)\n",
    "result_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From above result, we can see that latency is very close for different settings. The default setting (intra_op_num_threads=0, OMP_NUM_THREADS and OMP_WAIT_POLICY does not exist) performs the best. \n",
    "\n",
    "### Model Results Comparison Tool\n",
    "\n",
    "When a BERT model is optimized, some approximation is used in calculation. If your BERT model has three inputs, a script compare_bert_results.py can be used to do a quick verification. The tool will generate some fake input data, and compare the inference outputs of the original and optimized models. If outputs are all close, it is safe to use the optimized model.\n",
    "\n",
    "For GPU inference, the absolute or relative difference is larger than those numbers of CPU inference. Note that slight difference in output will not impact final result. We did end-to-end evaluation using SQuAD data set using a fine-tuned squad model, and F1 score is almost the same before/after optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100% passed for 100 random inputs given thresholds (rtol=0.01, atol=0.01).\r\n",
      "maximum absolute difference=1.9222497940063477e-06\r\n",
      "maximum relative difference=0.05027933046221733\r\n"
     ]
    }
   ],
   "source": [
    "!python -m onnxruntime_tools.transformers.compare_bert_results --baseline_model $export_model_path --optimized_model $optimized_fp32_model_path --batch_size 1 --sequence_length 128 --samples 100 --rtol 0.01 --atol 0.01 $GPU_OPTION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Model Optimization with Float16\n",
    "\n",
    "The optimizer.py script have an option **--float16** to convert model to use float16 to store weights. After the conversion, it could be faster to run in GPU with tensor cores like V100 or T4.\n",
    "\n",
    "Let's run tools to measure the performance on V100. The results show significant performance improvement: latency is about 3.4 ms for float32 model, and 1.8 ms for float16 model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "optimize_by_onnxruntime: Save optimized model by onnxruntime to ./onnx/bert-base-cased-squad_opset11_o1_cpu.onnx\n",
      "               apply: Fused LayerNormalization count: 25\n",
      "               apply: Fused Gelu count: 12\n",
      "               apply: Fused SkipLayerNormalization count: 25\n",
      "               apply: Fused Attention count: 12\n",
      "         prune_graph: Graph pruned: 0 inputs, 0 outputs and 5 nodes are removed\n",
      "               apply: Fused EmbedLayerNormalization(with mask) count: 1\n",
      "         prune_graph: Graph pruned: 0 inputs, 0 outputs and 12 nodes are removed\n",
      "         prune_graph: Graph pruned: 0 inputs, 0 outputs and 0 nodes are removed\n",
      "               apply: Fused BiasGelu count: 12\n",
      "               apply: Fused SkipLayerNormalization(add bias) count: 24\n",
      "            optimize: opset verion: 11\n",
      "  save_model_to_file: Output model to ./onnx/bert-base-cased-squad_opt_gpu_fp16.onnx\n",
      "get_fused_operator_statistics: Optimized operators:{'EmbedLayerNormalization': 1, 'Attention': 12, 'Gelu': 0, 'FastGelu': 0, 'BiasGelu': 12, 'LayerNormalization': 0, 'SkipLayerNormalization': 24}\n",
      "                main: The model has been fully optimized.\n"
     ]
    }
   ],
   "source": [
    "optimized_fp16_model_path = './onnx/bert-base-cased-squad_opt_{}_fp16.onnx'.format('gpu' if use_gpu else 'cpu')\n",
    "!python -m onnxruntime_tools.optimizer_cli --input $export_model_path --output $optimized_fp16_model_path --float16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test setting TestSetting(batch_size=1, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=None, omp_wait_policy=None, intra_op_num_threads=None, seed=3, verbose=False, inclusive=False, extra_latency=True)\n",
      "Generating 1000 samples for batch_size=1 sequence_length=128\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=0,OMP_NUM_THREADS=,OMP_WAIT_POLICY=,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 3.01 ms, Throughput = 331.90 QPS\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 3.12 ms, Throughput = 320.00 QPS\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=1,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 3.02 ms, Throughput = 331.39 QPS\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 3.01 ms, Throughput = 332.53 QPS\n",
      "skip duplicated test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "skip duplicated test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=1,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=1,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 3.04 ms, Throughput = 328.67 QPS\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 3.01 ms, Throughput = 331.72 QPS\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 3.04 ms, Throughput = 329.32 QPS\n",
      "Test summary is saved to onnx/perf_results_GPU_B1_S128_20200617-232234.txt\n"
     ]
    }
   ],
   "source": [
    "GPU_OPTION = '--use_gpu' if use_gpu else ''\n",
    "!python -m onnxruntime_tools.transformers.bert_perf_test --model $optimized_fp16_model_path --batch_size 1 --sequence_length 128 --samples 1000 --test_times 1 --inclusive --all $GPU_OPTION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Float32 model perf results from ./onnx/perf_results_GPU_B1_S128_20200617-232234.txt\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Latency(ms)</th>\n",
       "      <th>Latency_P50</th>\n",
       "      <th>Latency_P75</th>\n",
       "      <th>Latency_P90</th>\n",
       "      <th>Latency_P95</th>\n",
       "      <th>Latency_P99</th>\n",
       "      <th>Throughput(QPS)</th>\n",
       "      <th>intra_op_num_threads</th>\n",
       "      <th>OMP_NUM_THREADS</th>\n",
       "      <th>OMP_WAIT_POLICY</th>\n",
       "      <th>contiguous</th>\n",
       "      <th>warmup</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.01</td>\n",
       "      <td>2.79</td>\n",
       "      <td>2.81</td>\n",
       "      <td>2.86</td>\n",
       "      <td>5.08</td>\n",
       "      <td>7.16</td>\n",
       "      <td>332.53</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.01</td>\n",
       "      <td>2.80</td>\n",
       "      <td>2.81</td>\n",
       "      <td>2.88</td>\n",
       "      <td>4.52</td>\n",
       "      <td>7.05</td>\n",
       "      <td>331.90</td>\n",
       "      <td>0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.01</td>\n",
       "      <td>2.78</td>\n",
       "      <td>2.80</td>\n",
       "      <td>2.92</td>\n",
       "      <td>5.01</td>\n",
       "      <td>7.02</td>\n",
       "      <td>331.72</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.02</td>\n",
       "      <td>2.79</td>\n",
       "      <td>2.80</td>\n",
       "      <td>2.85</td>\n",
       "      <td>6.34</td>\n",
       "      <td>7.04</td>\n",
       "      <td>331.39</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>ACTIVE</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.04</td>\n",
       "      <td>2.80</td>\n",
       "      <td>2.82</td>\n",
       "      <td>2.93</td>\n",
       "      <td>5.56</td>\n",
       "      <td>7.08</td>\n",
       "      <td>329.32</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>PASSIVE</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>3.04</td>\n",
       "      <td>2.79</td>\n",
       "      <td>2.81</td>\n",
       "      <td>2.92</td>\n",
       "      <td>6.37</td>\n",
       "      <td>7.08</td>\n",
       "      <td>328.67</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>PASSIVE</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3.12</td>\n",
       "      <td>2.79</td>\n",
       "      <td>2.82</td>\n",
       "      <td>2.96</td>\n",
       "      <td>6.66</td>\n",
       "      <td>7.20</td>\n",
       "      <td>320.00</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>PASSIVE</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Latency(ms)  Latency_P50  Latency_P75  Latency_P90  Latency_P95  \\\n",
       "0         3.01         2.79         2.81         2.86         5.08   \n",
       "1         3.01         2.80         2.81         2.88         4.52   \n",
       "2         3.01         2.78         2.80         2.92         5.01   \n",
       "3         3.02         2.79         2.80         2.85         6.34   \n",
       "4         3.04         2.80         2.82         2.93         5.56   \n",
       "5         3.04         2.79         2.81         2.92         6.37   \n",
       "6         3.12         2.79         2.82         2.96         6.66   \n",
       "\n",
       "   Latency_P99  Throughput(QPS)  intra_op_num_threads OMP_NUM_THREADS  \\\n",
       "0         7.16           332.53                     1              12   \n",
       "1         7.05           331.90                     0                   \n",
       "2         7.02           331.72                    12              12   \n",
       "3         7.04           331.39                    12               1   \n",
       "4         7.08           329.32                    12              12   \n",
       "5         7.08           328.67                    12               1   \n",
       "6         7.20           320.00                     1              12   \n",
       "\n",
       "  OMP_WAIT_POLICY contiguous  warmup  \n",
       "0          ACTIVE       None    True  \n",
       "1                       None    True  \n",
       "2          ACTIVE       None    True  \n",
       "3          ACTIVE       None    True  \n",
       "4         PASSIVE       None    True  \n",
       "5         PASSIVE       None    True  \n",
       "6         PASSIVE       None    True  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import glob     \n",
    "import pandas\n",
    "latest_result_file = max(glob.glob(\"./onnx/perf_results_GPU_B1_S128_*.txt\"), key=os.path.getmtime)\n",
    "result_data = pandas.read_table(latest_result_file, converters={'OMP_NUM_THREADS': str, 'OMP_WAIT_POLICY':str})\n",
    "print(\"Float32 model perf results from\", latest_result_file)\n",
    "# Remove some columns that have same values for all rows.\n",
    "columns_to_remove = ['model', 'graph_optimization_level', 'batch_size', 'sequence_length', 'test_cases', 'test_times', 'use_gpu']\n",
    "result_data.drop(columns_to_remove, axis=1, inplace=True)\n",
    "result_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Throughput Tuning\n",
    "\n",
    "Some application need best throughput under some constraint on latency. This can be done by testing performance of different batch sizes. The tool could help on this.\n",
    "\n",
    "Here is an example that check the performance of multiple batch sizes (1, 2, 4, 8, 16, 32 and 64) using default settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test setting TestSetting(batch_size=32, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n",
      "Generating 1000 samples for batch_size=32 sequence_length=128\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=32,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 16.17 ms, Throughput = 1979.41 QPS\n",
      "test setting TestSetting(batch_size=1, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n",
      "Generating 1000 samples for batch_size=1 sequence_length=128\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 3.00 ms, Throughput = 333.83 QPS\n",
      "test setting TestSetting(batch_size=2, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n",
      "Generating 1000 samples for batch_size=2 sequence_length=128\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=2,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 3.59 ms, Throughput = 557.32 QPS\n",
      "test setting TestSetting(batch_size=64, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n",
      "Generating 1000 samples for batch_size=64 sequence_length=128\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=64,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 29.26 ms, Throughput = 2187.15 QPS\n",
      "test setting TestSetting(batch_size=4, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n",
      "Generating 1000 samples for batch_size=4 sequence_length=128\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=4,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 4.32 ms, Throughput = 926.92 QPS\n",
      "test setting TestSetting(batch_size=8, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n",
      "Generating 1000 samples for batch_size=8 sequence_length=128\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=8,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 6.32 ms, Throughput = 1266.63 QPS\n",
      "test setting TestSetting(batch_size=16, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n",
      "Generating 1000 samples for batch_size=16 sequence_length=128\n",
      "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=16,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n",
      "Average latency = 9.60 ms, Throughput = 1666.05 QPS\n",
      "Test summary is saved to onnx/perf_results_GPU_B1-2-4-8-16-32-64_S128_20200617-232401.txt\n"
     ]
    }
   ],
   "source": [
    "GPU_OPTION = '--use_gpu' if use_gpu else ''\n",
    "THREAD_SETTING = '--intra_op_num_threads 1 --omp_num_threads {} --omp_wait_policy ACTIVE'.format(psutil.cpu_count(logical=True))\n",
    "!python -m onnxruntime_tools.transformers.bert_perf_test --model $optimized_fp16_model_path --batch_size 1 2 4 8 16 32 64 --sequence_length 128 --samples 1000 --test_times 1 --inclusive $THREAD_SETTING $GPU_OPTION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Float16 model summary from ./onnx/perf_results_GPU_B1-2-4-8-16-32-64_S128_20200617-232401.txt\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Latency(ms)</th>\n",
       "      <th>Latency_P50</th>\n",
       "      <th>Latency_P75</th>\n",
       "      <th>Latency_P90</th>\n",
       "      <th>Latency_P95</th>\n",
       "      <th>Latency_P99</th>\n",
       "      <th>Throughput(QPS)</th>\n",
       "      <th>batch_size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.00</td>\n",
       "      <td>2.79</td>\n",
       "      <td>2.81</td>\n",
       "      <td>2.86</td>\n",
       "      <td>4.37</td>\n",
       "      <td>7.08</td>\n",
       "      <td>333.83</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.59</td>\n",
       "      <td>3.33</td>\n",
       "      <td>3.35</td>\n",
       "      <td>3.42</td>\n",
       "      <td>6.60</td>\n",
       "      <td>7.54</td>\n",
       "      <td>557.32</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.32</td>\n",
       "      <td>3.98</td>\n",
       "      <td>4.01</td>\n",
       "      <td>4.64</td>\n",
       "      <td>7.23</td>\n",
       "      <td>8.11</td>\n",
       "      <td>926.92</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6.32</td>\n",
       "      <td>5.94</td>\n",
       "      <td>5.97</td>\n",
       "      <td>7.61</td>\n",
       "      <td>8.96</td>\n",
       "      <td>10.12</td>\n",
       "      <td>1266.63</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9.60</td>\n",
       "      <td>9.22</td>\n",
       "      <td>9.25</td>\n",
       "      <td>11.32</td>\n",
       "      <td>12.33</td>\n",
       "      <td>13.34</td>\n",
       "      <td>1666.05</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>16.17</td>\n",
       "      <td>15.80</td>\n",
       "      <td>15.90</td>\n",
       "      <td>17.38</td>\n",
       "      <td>18.80</td>\n",
       "      <td>19.93</td>\n",
       "      <td>1979.41</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>29.26</td>\n",
       "      <td>28.89</td>\n",
       "      <td>29.01</td>\n",
       "      <td>30.63</td>\n",
       "      <td>32.53</td>\n",
       "      <td>33.28</td>\n",
       "      <td>2187.15</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Latency(ms)  Latency_P50  Latency_P75  Latency_P90  Latency_P95  \\\n",
       "0         3.00         2.79         2.81         2.86         4.37   \n",
       "1         3.59         3.33         3.35         3.42         6.60   \n",
       "2         4.32         3.98         4.01         4.64         7.23   \n",
       "3         6.32         5.94         5.97         7.61         8.96   \n",
       "4         9.60         9.22         9.25        11.32        12.33   \n",
       "5        16.17        15.80        15.90        17.38        18.80   \n",
       "6        29.26        28.89        29.01        30.63        32.53   \n",
       "\n",
       "   Latency_P99  Throughput(QPS)  batch_size  \n",
       "0         7.08           333.83           1  \n",
       "1         7.54           557.32           2  \n",
       "2         8.11           926.92           4  \n",
       "3        10.12          1266.63           8  \n",
       "4        13.34          1666.05          16  \n",
       "5        19.93          1979.41          32  \n",
       "6        33.28          2187.15          64  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import glob     \n",
    "import pandas\n",
    "latest_result_file = max(glob.glob(\"./onnx/perf_results_*.txt\"), key=os.path.getmtime)\n",
    "result_data = pandas.read_table(latest_result_file, converters={'OMP_NUM_THREADS': str, 'OMP_WAIT_POLICY':str})\n",
    "print(\"Float16 model summary from\", latest_result_file)\n",
    "columns_to_remove = ['model', 'graph_optimization_level', 'test_cases', 'test_times', 'use_gpu', 'warmup', 'sequence_length']\n",
    "columns_to_remove.extend(['intra_op_num_threads', 'OMP_NUM_THREADS', 'OMP_WAIT_POLICY', 'contiguous'])\n",
    "result_data.drop(columns_to_remove, axis=1, inplace=True)\n",
    "result_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Additional Info\n",
    "\n",
    "Note that running Jupyter Notebook has significant impact on performance result. You can close Jupyter Notebook and other applications, then run the performance test in a console to get more accurate performance numbers.\n",
    "\n",
    "We have a [benchmark script](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/run_benchmark.sh). It is recommended to use it measure inference speed of OnnxRuntime.\n",
    "\n",
    "[OnnxRuntime C API](https://github.com/microsoft/onnxruntime/blob/master/docs/C_API.md) could get slightly better performance than python API. If you use C API in inference, you can use OnnxRuntime_Perf_Test.exe built from source to measure performance instead.\n",
    "\n",
    "Here is the machine configuration that generated the above results. You might get slower or faster result according to your hardware."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"gpu\": {\r\n",
      "    \"driver_version\": \"440.64.00\",\r\n",
      "    \"devices\": [\r\n",
      "      {\r\n",
      "        \"memory_total\": 16945512448,\r\n",
      "        \"memory_available\": 14110883840,\r\n",
      "        \"name\": \"Tesla V100-PCIE-16GB\"\r\n",
      "      },\r\n",
      "      {\r\n",
      "        \"memory_total\": 16945512448,\r\n",
      "        \"memory_available\": 16932601856,\r\n",
      "        \"name\": \"Tesla V100-PCIE-16GB\"\r\n",
      "      }\r\n",
      "    ]\r\n",
      "  },\r\n",
      "  \"cpu\": {\r\n",
      "    \"brand\": \"Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz\",\r\n",
      "    \"cores\": 12,\r\n",
      "    \"logical_cores\": 12,\r\n",
      "    \"hz\": \"2.5940 GHz\",\r\n",
      "    \"l2_cache\": \"256 KB\",\r\n",
      "    \"l3_cache\": \"35840 KB\",\r\n",
      "    \"processor\": \"x86_64\"\r\n",
      "  },\r\n",
      "  \"memory\": {\r\n",
      "    \"total\": 236645588992,\r\n",
      "    \"available\": 222567559168\r\n",
      "  },\r\n",
      "  \"python\": \"3.7.7.final.0 (64 bit)\",\r\n",
      "  \"os\": \"Linux-4.15.0-1089-azure-x86_64-with-debian-stretch-sid\",\r\n",
      "  \"onnxruntime\": {\r\n",
      "    \"version\": \"1.3.0\",\r\n",
      "    \"support_gpu\": true\r\n",
      "  },\r\n",
      "  \"pytorch\": {\r\n",
      "    \"version\": \"1.5.0\",\r\n",
      "    \"support_gpu\": true\r\n",
      "  },\r\n",
      "  \"tensorflow\": null\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!{sys.executable} -m onnxruntime_tools.transformers.machine_info --silent"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PyCharm (ccks_ner-master)",
   "language": "python",
   "name": "pycharm-de4c0941"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
